home *** CD-ROM | disk | FTP | other *** search
- Subject: v23i040: Flex, a fast lex replacement, Part04/10
- Newsgroups: comp.sources.unix
- Approved: rsalz@uunet.UU.NET
- X-Checksum-Snefru: a7d25fc5 420511c0 bf5c3f37 a812d241
-
- Submitted-by: Vern Paxson <vern@cs.cornell.edu>
- Posting-number: Volume 23, Issue 40
- Archive-name: flex2.3/part04
-
- #! /bin/sh
- # This is a shell archive. Remove anything before this line, then feed it
- # into a shell via "sh file" or similar. To overwrite existing files,
- # type "sh file -c".
- # The tool that generated this appeared in the comp.sources.unix newsgroup;
- # send mail to comp-sources-unix@uunet.uu.net if you want that tool.
- # Contents: MISC/Atari.patches flex.1
- # Wrapped by rsalz@litchi.bbn.com on Wed Oct 10 13:24:00 1990
- PATH=/bin:/usr/bin:/usr/ucb ; export PATH
- echo If this archive is complete, you will see the following message:
- echo ' "shar: End of archive 4 (of 10)."'
- if test -f 'MISC/Atari.patches' -a "${1}" != "-c" ; then
- echo shar: Will not clobber existing file \"'MISC/Atari.patches'\"
- else
- echo shar: Extracting \"'MISC/Atari.patches'\" \(29474 characters\)
- sed "s/^X//" >'MISC/Atari.patches' <<'END_OF_FILE'
- X(Message inbox:32)
- XDate: Mon, 03 Jul 89 21:15:32 CET
- XFrom: V61%DHDURZ1.BITNET@lbl.gov
- XSubject: Flex, bug fix, improvments, patches for Minix & TOS
- XTo: vern@lbl-csam.arpa
- X
- XAt first I have to thank you for your wonderful program. I had ported the
- Xold version to OS9,TOS (Atari ST) and Minix and the new version 2.1 Beta
- Xto Minix and TOS.
- X
- XWhile porting and using flex I detected a bug and made some improvements.
- XI have included a shared, compressed and uuencoded file contaning all cdiffs
- Xand additional files (Sorry, but I'm on EBCDIC-Bitnet) and a short discussion
- Xof the changes. Even some of the TOS specific changes might be of general
- Xinterest !
- X
- XI posted these cdiffs to the minix discussion group, but I think it's up
- Xto you to post them to the unix-sources group. If you plan to post even
- Xthe TOS compiler specific patches please contact me because there might be
- Xfurther compiler (P.D.) additions. If you have an interest I could also
- Xport the new version to OS9 -- this is a little bit more difficult, because
- XOS9 uses CR as end of line character (the EOL char. is coded into the
- Xinitscan.c tables,...). It is necessary to change all occurences of '\n' to
- Xmacros and variables and it's useful to add a new -n options (see commented
- Xline in main.c)
- X
- X
- X
- XThe changes: (1.7.89 RAL)
- X
- X - Bug fix: The original flex didn't like trailing spaces in exclusive start
- X condition lists ! If you add an trailing space to line 68 in scan.l
- X
- X "%x SECT2 SECT2PROLOG SECT3 CODEBLOCK PICKUPDEF SC CARETISBOL NUM QUOTE "
- X
- X you get a misleading error message:
- X
- X "Syntax error at line 69: bad start condition list"
- X
- X This bug can either be fixed in parse.y or in scan.l . I have chosen the
- X last because there the fix is minimal: Just change the rule (line 128)
- X
- X "\n" to "{OPTWS}\n"
- X
- X - Enhancements:
- X - new option "-?" that provides some help information about the other
- X flags (main.c)
- X - new option "-aTMPPATH" that allows a redefinition of the standard
- X path for the temporary file (this might be rather large >200k if
- X F,f options are selected). (main.c, flexdef.h (l.376))
- X - hexdump of illegal characters -- this proved to be a useful debugging
- X tool especialy if invisible control characters occur which weren't
- X covered by the rules. (scan.l fprintf statement line 129,...)
- X
- X - Patches due to TOS
- X - General: TOS uses CR,LF as end of line char., Flex wants only a single
- X LF as EOL char. Therefore all I/O must be translated using f* calls.
- X This is done everywhere besides the YY_INPUT macro (flex.skel (scan.c),
- X line 31) that uses a low level 'read'. This should be definitly changed
- X to fread, so that all I/O calls occur on the same level.
- X - the short action_file_name has been "flexXXXXXX.tmp", but that's too
- X much for TOS,MSDOS ! I changed it to "flexXXXX.tmp" in main.c (patch
- X covered by the -a option additions)
- X - some compilers don't like external names that are ambiguous within
- X the first 8 characters. I defined macros that change all these long
- X external names to names that are unique within the first 8 characters.
- X Just define SHORT_EXTERNAL_NAMES to use this feature (flexdef.h)
- X - some statement changes that some compiler don't like:
- X typedef *xxx[] -> typedef **xxx (flexdef.h.,l.308)
- X "/*" -> '/','*' within a comment in (scan.l, l.209)
- X - changed short "lexyy.c" to "lex_yy.c" what's more similar to the unix
- X "lex.yy.c" (main.c).
- X - a few further really compiler dependent changes provided with
- X #ifdef ATARI && LATTICE res. TURBO braces.
- X
- X - Additional TOS only files
- X - Makefile.tos: Common makefile for all TOS compilers. If you add further
- X ones please email me the new makefile.
- X - flex.lnk: Lattice - GST linker argument extension file
- X - flex.tlk: Turbo linker argument extension file
- X
- X
- XAdditional remarks:
- X
- XI didn't add a new initscan.c (= flex -ist scan.l). The current one is good
- Xenough for a first compilation. With this first version of flex you can
- Xrebuild your own scan.c and the final flex version !
- X
- XMinix ST :
- X - I had to "chmem =70000 cv" (>50000) to get flex linked
- X - 'memset' (PC 1.3, EFTH40,...) is necessary
- X - chmem =90000 flex may be sufficient
- X
- XMinix PC :
- X It should be possible to port Flex to Minix PC. The current sizes of flex
- X are:
- X Minix ST (ACK) Lattice (TOS) Turbo (TOS)
- X
- X size 75300 83305 57957
- X compilation time 22' 15' 3'40"
- X flex -is scan.l 1'49" 43" 30"
- X
- X The Minix ST size includes the bad generated code using only a subset of
- X the 68000 commands, long addresses only and a huge relocation table.
- X Therefore the PC size will be <64 k ! More serious is the fact that I had
- X to chmem =90000 flex to get scan.l converted to scan.c . But I never saw
- X a more complex lex source than scan.l -- so it should be possible to
- X reduce some array sizes without limitation for all day usage.
- X
- X No one volunteered yet for a Minix PC port -- but if someone will try it
- X I would provide him with a new scan.c and some hints.
- X
- XTOS:
- X Don't forget to adapt the flexskel path within flexdef.h !
- X
- X
- XBitnet: V61@DHDURZ1 Ronald Lamprecht
- XUUCP: ...!unido!DHDURZ1.bitnet!V61 Theoretische Physik
- XARPAnet: V61%DHDURZ1.BITNET@CUNYVM.CUNY.EDU (Heidelberg, West Germany)
- X(Message inbox:36)
- XDate: Wed, 05 Jul 89 21:16:07 CET
- XFrom: V61%DHDURZ1.BITNET@csa2.lbl.gov
- XSubject: Re: What is TOS
- XTo: vern@lbl-csam.arpa
- X
- XTOS is the name of the Atari ST operating system that is very similar
- Xto MSDOS (Both use CR,LF as end of line character). Therefore I have
- Xbeen astonished that no EOL convertion porblems occur within MSDOS.
- X
- XI have been aware of the double buffering when changing read to fread and I
- Xaccept your argument of a possible slow down. But if you integrate the other
- XAtari - TOS changes, please insert a
- X #ifdef ATARI
- X fread ....
- X #else
- X read ....
- X #endif
- Xin flex.skel .
- X
- XBitnet: V61@DHDURZ1 Ronald Lamprecht
- XUUCP: ...!unido!DHDURZ1.bitnet!V61 Theoretische Physik
- XARPAnet: V61%DHDURZ1.BITNET@CUNYVM.CUNY.EDU (Heidelberg, West Germany)
- X
- X
- X
- X
- Xecho x - Makefile_cdiff
- Xsed '/^X/s///' > Makefile_cdiff << '/'
- XX*** Src_2.1/Makefile Thu Jun 28 00:06:42 1989
- XX--- Makefile Thu Jul 3 02:12:48 1989
- XX***************
- XX*** 5,10 ****
- XX--- 5,11 ----
- XX # Porting considerations:
- XX #
- XX # For System V Unix machines, add -DSYS_V to CFLAGS.
- XX+ # For Minix (ST), add -DSYS_V to CFLAGS
- XX # For Vax/VMS, add -DSYS_V to CFLAGS.
- XX # For MS-DOS, add "-DMS_DOS -DSYS_V" to CFLAGS. Create \tmp if not present.
- XX # You will also want to rename flex.skel to something with a three
- XX***************
- XX*** 21,28 ****
- XX SKELETON_DIR = /usr/local/lib
- XX SKELETON_FILE = flex.skel
- XX SKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
- XX! CFLAGS = -O
- XX! LDFLAGS = -s
- XX
- XX FLEX_FLAGS =
- XX FLEX = ./flex
- XX--- 22,29 ----
- XX SKELETON_DIR = /usr/local/lib
- XX SKELETON_FILE = flex.skel
- XX SKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
- XX! CFLAGS = -O -DSYS_V
- XX! LDFLAGS =
- XX
- XX FLEX_FLAGS =
- XX FLEX = ./flex
- XX***************
- XX*** 57,63 ****
- XX yylex.c
- XX
- XX flex : $(FLEXOBJS)
- XX! $(CC) $(CFLAGS) -o flex $(LDFLAGS) $(FLEXOBJS)
- XX
- XX first_flex:
- XX cp initscan.c scan.c
- XX--- 58,65 ----
- XX yylex.c
- XX
- XX flex : $(FLEXOBJS)
- XX! $(CC) $(CFLAGS) -o flex $(FLEXOBJS) $(LDFLAGS)
- XX! chmem =150000 flex
- XX
- XX first_flex:
- XX cp initscan.c scan.c
- X/
- Xecho x - flex.skel_cdif
- Xsed '/^X/s///' > flex.skel_cdif << '/'
- XX*** Src_2.1/flex.skel Thu Jun 28 00:19:20 1989
- XX--- flex.skel Thu Jul 2 22:18:31 1989
- XX***************
- XX*** 28,34 ****
- XX * is returned in "result".
- XX */
- XX #define YY_INPUT(buf,result,max_size) \
- XX! if ( (result = read( fileno(yyin), buf, max_size )) < 0 ) \
- XX YY_FATAL_ERROR( "read() in flex scanner failed" );
- XX #define YY_NULL 0
- XX #define yyterminate() return ( YY_NULL )
- XX--- 28,34 ----
- XX * is returned in "result".
- XX */
- XX #define YY_INPUT(buf,result,max_size) \
- XX! if ( (result = fread( buf,1, max_size, yyin )) < 0 ) \
- XX YY_FATAL_ERROR( "read() in flex scanner failed" );
- XX #define YY_NULL 0
- XX #define yyterminate() return ( YY_NULL )
- X/
- Xecho x - flexdef.h_cdif
- Xsed '/^X/s///' > flexdef.h_cdif << '/'
- XX*** Src_2.1/flexdef.h Thu Jun 28 00:43:27 1989
- XX--- flexdef.h Thu Jul 3 02:45:50 1989
- XX***************
- XX*** 26,31 ****
- XX--- 26,40 ----
- XX
- XX /* @(#) $Header: flexdef.h,v 2.0 89/06/20 15:49:50 vern Locked $ (LBL) */
- XX
- XX+ #ifdef ATARI
- XX+ #define SYS_V
- XX+ #define abs(x) ((x) < 0 ? -(x) : (x))
- XX+ #define SHORT_FILE_NAMES
- XX+ #ifdef TURBO
- XX+ #define SHORT_EXTERNAL_NAMES
- XX+ #endif
- XX+ #endif
- XX+
- XX #ifndef FILE
- XX #include <stdio.h>
- XX #endif
- XX***************
- XX*** 41,47 ****
- XX #endif
- XX
- XX #ifndef VMS
- XX! char *memset();
- XX #else
- XX /* memset is needed for old versions of the VMS C runtime library */
- XX #define memset(s, c, n) \
- XX--- 50,58 ----
- XX #endif
- XX
- XX #ifndef VMS
- XX! #ifndef ATARI && TURBO
- XX! char *memset();
- XX! #endif
- XX #else
- XX /* memset is needed for old versions of the VMS C runtime library */
- XX #define memset(s, c, n) \
- XX***************
- XX*** 81,91 ****
- XX--- 92,129 ----
- XX #define true 1
- XX #define false 0
- XX
- XX+ #ifdef ATARI
- XX+ #define DEFAULT_SKELETON_FILE "D:\\include\\flexskel"
- XX+ #endif
- XX+
- XX
- XX #ifndef DEFAULT_SKELETON_FILE
- XX #define DEFAULT_SKELETON_FILE "flex.skel"
- XX #endif
- XX
- XX+ #ifdef SHORT_EXTERNAL_NAMES
- XX+ /* avoid long external names that are ambiguous within the first 8 characters */
- XX+ #define current_mns c__mns
- XX+ #define current_max_rules c__max_rules
- XX+ #define current_max_state_type c__max_state_type
- XX+ #define current_max_scs c__max_scs
- XX+ #define current_max_dfa_size c__max__size
- XX+ #define current_max_xpairs c__max_xpairs
- XX+ #define current_max_template_xpairs c__max_template_xpairs
- XX+ #define current_max_dfas c__max_dfas
- XX+ #define current_maxccls c__maxccles
- XX+ #define current_max_ccl_tbl_size c__max_ccl_tbl_size
- XX+ #define indent_puts ind_puts
- XX+ #define indent_put2s ind_put2s
- XX+ #define gen_next_compressed_state gen_n_comressed_state
- XX+ #define gen_next_match gen_n_match
- XX+ #define gen_next_state gen_n_state
- XX+ #define variable_trailing_context_rules var_tr_context_rules
- XX+ #define variable_trailing_rule var_tr_rule
- XX+ #define backtrack_report backtr_report
- XX+ #define backtrack_file backtr_file
- XX+ #endif
- XX+
- XX /* special chk[] values marking the slots taking by end-of-buffer and action
- XX * numbers
- XX */
- XX***************
- XX*** 305,311 ****
- XX int int_val;
- XX } ;
- XX
- XX! typedef struct hash_entry *hash_table[];
- XX
- XX #define NAME_TABLE_HASH_SIZE 101
- XX #define START_COND_HASH_SIZE 101
- XX--- 343,349 ----
- XX int int_val;
- XX } ;
- XX
- XX! typedef struct hash_entry **hash_table;
- XX
- XX #define NAME_TABLE_HASH_SIZE 101
- XX #define START_COND_HASH_SIZE 101
- XX***************
- XX*** 372,378 ****
- XX extern int datapos, dataline, linenum;
- XX extern FILE *skelfile, *yyin, *temp_action_file, *backtrack_file;
- XX extern char *infilename;
- XX! extern char action_file_name[];
- XX
- XX
- XX /* variables for stack of states having only one out-transition:
- XX--- 410,416 ----
- XX extern int datapos, dataline, linenum;
- XX extern FILE *skelfile, *yyin, *temp_action_file, *backtrack_file;
- XX extern char *infilename;
- XX! extern char *action_file_name;
- XX
- XX
- XX /* variables for stack of states having only one out-transition:
- X/
- Xecho x - main.c_cdiff
- Xsed '/^X/s///' > main.c_cdiff << '/'
- XX*** Src_2.1/main.c Thu Jun 28 00:30:39 1989
- XX--- main.c Thu Jul 3 02:27:47 1989
- XX***************
- XX*** 81,96 ****
- XX FILE *temp_action_file;
- XX FILE *backtrack_file;
- XX int end_of_buffer_state;
- XX! #ifndef SHORT_FILE_NAMES
- XX! char action_file_name[] = "/tmp/flexXXXXXX";
- XX! #else
- XX! char action_file_name[] = "flexXXXXXX.tmp";
- XX! #endif
- XX!
- XX #ifndef SHORT_FILE_NAMES
- XX static char outfile[] = "lex.yy.c";
- XX #else
- XX! static char outfile[] = "lexyy.c";
- XX #endif
- XX static int outfile_created = 0;
- XX
- XX--- 81,91 ----
- XX FILE *temp_action_file;
- XX FILE *backtrack_file;
- XX int end_of_buffer_state;
- XX! char *action_file_name;
- XX #ifndef SHORT_FILE_NAMES
- XX static char outfile[] = "lex.yy.c";
- XX #else
- XX! static char outfile[] = "lex_yy.c";
- XX #endif
- XX static int outfile_created = 0;
- XX
- XX***************
- XX*** 328,333 ****
- XX--- 323,329 ----
- XX {
- XX int i, sawcmpflag, use_stdout;
- XX char *arg, *skelname = NULL, *flex_gettime(), clower(), *mktemp();
- XX+ char *tmp_action =(char *)0, *malloc();
- XX
- XX printstats = syntaxerror = trace = spprdflt = interactive = caseins = false;
- XX backtrack_report = performance_report = ddebug = fulltbl = fullspd = false;
- XX***************
- XX*** 349,354 ****
- XX--- 345,355 ----
- XX for ( i = 1; arg[i] != '\0'; ++i )
- XX switch ( arg[i] )
- XX {
- XX+ case 'a':
- XX+ if ( i != 1 )
- XX+ flexerror( "-a flag must be given separately" );
- XX+ tmp_action = &arg[i+1];
- XX+ goto get_next_arg;
- XX case 'b':
- XX backtrack_report = true;
- XX break;
- XX***************
- XX*** 445,452 ****
- XX printstats = true;
- XX break;
- XX
- XX! default:
- XX! lerrif( "unknown flag %c", (int) arg[i] );
- XX break;
- XX }
- XX
- XX--- 446,458 ----
- XX printstats = true;
- XX break;
- XX
- XX! case '?':
- XX! flexinfo(0);
- XX! break;
- XX!
- XX! default:
- XX! fprintf(stderr,"flex : unknown flag %c\n", (int) arg[i] );
- XX! flexinfo(1);
- XX break;
- XX }
- XX
- XX***************
- XX*** 454,459 ****
- XX--- 460,493 ----
- XX ;
- XX }
- XX
- XX+
- XX+ /* if you change the default tmp file names don't forget to change the
- XX+ initialization for i, too !
- XX+
- XX+ */
- XX+ #ifndef SHORT_FILE_NAMES
- XX+ i = 10;
- XX+ if (!tmp_action) i += 5;
- XX+ #else
- XX+ i = 12;
- XX+ #endif
- XX+ if (tmp_action)
- XX+ i += strlen(tmp_action) + 1;
- XX+ if((action_file_name = malloc(i+1)) == (char *)0)
- XX+ flexerror("No memory for action_file_name");
- XX+ *action_file_name = (char) NULL;
- XX+ if (tmp_action)
- XX+ strcat(action_file_name,tmp_action);
- XX+ #ifndef SHORT_FILE_NAMES
- XX+ else
- XX+ strcat(action_file_name,"/tmp");
- XX+ strcat(action_file_name,"/flexXXXXXX");
- XX+ #else
- XX+ if (tmp_action)
- XX+ strcat(action_file_name,"/");
- XX+ strcat(action_file_name,"flexXXXX.tmp");
- XX+ #endif
- XX+
- XX if ( (fulltbl || fullspd) && usemecs )
- XX flexerror( "full table and -cm don't make sense together" );
- XX
- XX***************
- XX*** 520,526 ****
- XX if ( (skelfile = fopen( skelname, "r" )) == NULL )
- XX lerrsf( "can't open skeleton file %s", skelname );
- XX
- XX! (void) mktemp( action_file_name );
- XX
- XX if ( (temp_action_file = fopen( action_file_name, "w" )) == NULL )
- XX lerrsf( "can't open temporary action file %s", action_file_name );
- XX--- 554,562 ----
- XX if ( (skelfile = fopen( skelname, "r" )) == NULL )
- XX lerrsf( "can't open skeleton file %s", skelname );
- XX
- XX! #ifndef ATARI && LATTICE
- XX! (void) mktemp( action_file_name );
- XX! #endif
- XX
- XX if ( (temp_action_file = fopen( action_file_name, "w" )) == NULL )
- XX lerrsf( "can't open temporary action file %s", action_file_name );
- XX***************
- XX*** 566,571 ****
- XX--- 602,640 ----
- XX }
- XX
- XX
- XX+ flexinfo(status)
- XX+ int status;
- XX+ {
- XX+ fprintf(stderr,"Syntax : FLEX inp_file\n");
- XX+ fprintf(stderr,"Function: fast lexical analyzer generator V%s\n",flex_version);
- XX+ fprintf(stderr,"Options : a dir_path : directory path for temporary files\n");
- XX+ fprintf(stderr," - b : generate backtracking information to lex.backtrack\n");
- XX+ fprintf(stderr," - c : compressed table, no equiv., no meta equiv.classes\n");
- XX+ fprintf(stderr," e : equivalence classes\n");
- XX+ fprintf(stderr," F : fast table\n");
- XX+ fprintf(stderr," |f : full table\n");
- XX+ fprintf(stderr," |m : meta equivalence classes\n");
- XX+ fprintf(stderr," - d : generate debugging scanner\n");
- XX+ fprintf(stderr," - F : fast table\n");
- XX+ fprintf(stderr," - f : full (not compressed) table\n");
- XX+ fprintf(stderr," - I : generate interactive scanner\n");
- XX+ fprintf(stderr," - i : generate case-insensitive scanner\n");
- XX+ fprintf(stderr," - L : supress #line directives\n");
- XX+ /* fprintf(stderr," - n hexnum : generate scanner using <hexnum> as newline char.\n");*/
- XX+ fprintf(stderr," - p : generate performance report to stderr\n");
- XX+ fprintf(stderr," - S skeleton_path : file path for skeleton file\n");
- XX+ fprintf(stderr," - s : suppress echo of unmatched scanner input to stdout\n");
- XX+ fprintf(stderr," - T : run flex in trace mode\n");
- XX+ #ifdef ATARI
- XX+ fprintf(stderr," - t : place result on stdout instead of lex_yy.c\n");
- XX+ #else
- XX+ fprintf(stderr," - t : place result on stdout instead of lex.yy.c\n");
- XX+ #endif
- XX+ fprintf(stderr," - v : print statistics of generated scanner\n");
- XX+ fprintf(stderr," default = -cem\n");
- XX+ exit(status);
- XX+ }
- XX+
- XX /* readin - read in the rules section of the input file(s)
- XX *
- XX * synopsis
- X/
- Xecho x - scan.l_cdiff
- Xsed '/^X/s///' > scan.l_cdiff << '/'
- XX*** Src_2.1/scan.l Thu Jun 30 19:42:00 1989
- XX--- scan.l Thu Jul 2 22:19:26 1989
- XX***************
- XX*** 125,132 ****
- XX
- XX {SCNAME} RETURNNAME;
- XX ^{OPTWS}\n ++linenum; /* allows blank lines in section 1 */
- XX! \n ++linenum; return ( '\n' );
- XX! . synerr( "illegal character" ); BEGIN(RECOVER);
- XX
- XX
- XX <C_COMMENT>"*/" ECHO; BEGIN(0);
- XX--- 125,136 ----
- XX
- XX {SCNAME} RETURNNAME;
- XX ^{OPTWS}\n ++linenum; /* allows blank lines in section 1 */
- XX! {OPTWS}\n ++linenum; return ( '\n' );
- XX! . {
- XX! synerr( "illegal character" );
- XX! fprintf(stderr,"Char : $%x\n",yytext[yyleng-1]);
- XX! BEGIN(RECOVER);
- XX! }
- XX
- XX
- XX <C_COMMENT>"*/" ECHO; BEGIN(0);
- XX***************
- XX*** 206,212 ****
- XX <SECT2>^{OPTWS}\n ++linenum; /* allow blank lines in section 2 */
- XX
- XX /* this horrible mess of a rule matches indented lines which
- XX! * do not contain "/*". We need to make the distinction because
- XX * otherwise this rule will be taken instead of the rule which
- XX * matches the beginning of comments like this one
- XX */
- XX--- 210,216 ----
- XX <SECT2>^{OPTWS}\n ++linenum; /* allow blank lines in section 2 */
- XX
- XX /* this horrible mess of a rule matches indented lines which
- XX! * do not contain '/','*'. We need to make the distinction because
- XX * otherwise this rule will be taken instead of the rule which
- XX * matches the beginning of comments like this one
- XX */
- X/
- Xecho x - Makefile.tos
- Xsed '/^X/s///' > Makefile.tos << '/'
- XX# make file for "flex" tool
- XX
- XX# @(#) $Header: Makefile,v 2.3 89/06/20 17:27:12 vern Exp $ (LBL)
- XX
- XX# Porting considerations:
- XX#
- XX# For System V Unix machines, add -DSYS_V to CFLAGS.
- XX# For Vax/VMS, add -DSYS_V to CFLAGS.
- XX# For MS-DOS, add "-DMS_DOS -DSYS_V" to CFLAGS. Create \tmp if not present.
- XX# You will also want to rename flex.skel to something with a three
- XX# character extension, change SKELETON_FILE below appropriately,
- XX# For Amiga, add "-DAMIGA -DSYS_V" to CFLAGS.
- XX#
- XX# A long time ago, flex was successfully built using Microsoft C and
- XX# the following options: /AL, /stack:10000, -LARGE, -Ml, -Mt128, -DSYS_V
- XX
- XX
- XX# the first time around use "make first_flex"
- XX
- XX# The following definitions must be set according to your compiler -
- XX# examples for a Lattice Compiler with GST assembler and TURBO C with
- XX# assembler are provided below and must just be updated (don't forget to
- XX# update the linker argument extension files (*.lnk,*.tlk), too) :
- XX#
- XX#CCPATH = path to compiler directory without trailing \
- XX#CHPATH = path to header files without trailing \
- XX#CC = filename of the compiler
- XX#CFLAGS = compiler option flags
- XX#CIEXT = extension of C sources that should be used for input filenames
- XX#ASMPATH = path to assembler directory without trailing \
- XX#ASM = filename of the assembler
- XX#AFLAGS = assembler option flags
- XX#AIEXT = extension of assembler sources that should be used for assembler
- XX# input filenames
- XX#AEXT = general assembler filename extension
- XX#LNKPATH = path to linker directory without trailing \
- XX#LINK = filename of the linker
- XX#LFLAG0 = first option (full pathname of C startupcode)
- XX#LFLAG1 = further options + option flag for argument extension filename
- XX#LFLAG2 = further options + option flag for output-filename
- XX#LNKEXT = extension of linker argument extension file
- XX#OIEXT = extension of objects that should be used for linker input files
- XX#OEXT = general object file extension
- XX
- XX# Lattice definitions
- XXCCPATH = d:\latt
- XXCHPATH = d:\latt\include
- XXCC = lc.ttp
- XXCFLAGS = -h -n -cw -cc -i$(CHPATH)\ -g$(CCPATH)\ -dLATTICE -dATARI
- XXCIEXT =
- XXASMPATH = d:\gst
- XXASM = assem.ttp
- XXAFLAGS = -nolist -errors errors.out
- XXAIEXT =
- XXAEXT = .asm
- XXLNKPATH = d:\gst
- XXLINK = ld.ttp
- XXLFLAG0 =
- XXLFLAG1 = -with
- XXLFLAG2 = -nolist -sec -mem 200 -prog
- XXLNKEXT = .lnk
- XXOIEXT =
- XXOEXT = .bin
- XX
- XX# Turbo definitions
- XX#CCPATH = d:\turbo
- XX#CHPATH = d:\turbo\include
- XX#CC = tcc.prg
- XX#CFLAGS = -GJMPRZ -H=$(CHPATH)\ -w- -DTURBO -DATARI
- XX#CIEXT = .c
- XX#ASMPATH = d:\turbo
- XX#ASM = mas.prg
- XX#AFLAGS =
- XX#AIEXT = .s
- XX#AEXT = .s
- XX#LNKPATH = d:\turbo
- XX#LINK = tlink.ttp
- XX#LFLAG0 = $(LNKPATH)\lib\tcstart.o
- XX#LFLAG1 = -I=
- XX#LFLAG2 = -O=
- XX#LNKEXT = .tlk
- XX#OIEXT = .o
- XX#OEXT = .o
- XX
- XX# Other definitions
- XX# (not used for Atari because of short argument string - defined in flexdef.h
- XX
- XXSKELETON_DIR = /usr/lib
- XXSKELETON_FILE = flex.skel
- XXSKELFLAGS = -DDEFAULT_SKELETON_FILE=\"$(SKELETON_DIR)/$(SKELETON_FILE)\"
- XX
- XX
- XXFLEX = d:\etc\flex.ttp
- XXFLEX_FLAGS = -is
- XXYACC = d:\etc\yacc.ttp
- XXRENAME = d:\bin\rename
- XXDEL = d:\bin\del
- XX
- XX# Internal definitions
- XXLNK = $(LNKPATH)\$(LINK)
- XX
- XXFLEXOBJS = \
- XX ccl$(OEXT) \
- XX dfa$(OEXT) \
- XX ecs$(OEXT) \
- XX gen$(OEXT) \
- XX main$(OEXT) \
- XX misc$(OEXT) \
- XX nfa$(OEXT) \
- XX parse$(OEXT) \
- XX scan$(OEXT) \
- XX sym$(OEXT) \
- XX tblcmp$(OEXT) \
- XX yylex$(OEXT)
- XX
- XXFLEX_C_SOURCES = \
- XX ccl.c \
- XX dfa.c \
- XX ecs.c \
- XX gen.c \
- XX main.c \
- XX misc.c \
- XX nfa.c \
- XX parse.c \
- XX scan.c \
- XX sym.c \
- XX tblcmp.c \
- XX yylex.c
- XX
- XXflex : $(FLEXOBJS)
- XX $(LNK) $(LFLAG0) $(LFLAG1)flex$(LNKEXT) $(LFLAG2)flex.ttp
- XX
- XXfirst_flex:
- XX cp initscan.c scan.c
- XX make $(MFLAGS) flex
- XX
- XXparse.h parse.c : parse.y
- XX $(YACC) -d parse.y
- XX $(DEL) parse.c
- XX $(RENAME) y_tab.c parse.c
- XX $(DEL) parse.h
- XX $(RENAME) y_tab.h parse.h
- XX
- XXscan.c : scan.l
- XX $(FLEX) $(FLEX_FLAGS) scan.l
- XX $(RENAME) lex_yy.c scan.c
- XX
- XXscan$(OEXT) : scan.c parse.h flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) scan$(CIEXT)
- XX
- XXmain$(OEXT) : main.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) main$(CIEXT)
- XX
- XXccl$(OEXT) : ccl.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) ccl$(CIEXT)
- XX
- XXdfa$(OEXT) : dfa.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) dfa$(CIEXT)
- XX
- XXecs$(OEXT) : ecs.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) ecs$(CIEXT)
- XX
- XXgen$(OEXT) : gen.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) gen$(CIEXT)
- XX
- XXmisc$(OEXT) : misc.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) misc$(CIEXT)
- XX
- XXnfa$(OEXT) : nfa.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) nfa$(CIEXT)
- XX
- XXparse$(OEXT) : parse.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) parse$(CIEXT)
- XX
- XXsym$(OEXT) : sym.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) sym$(CIEXT)
- XX
- XXtblcmp$(OEXT) : tblcmp.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) tblcmp$(CIEXT)
- XX
- XXyylex$(OEXT) : yylex.c flexdef.h
- XX $(CCPATH)\$(CC) $(CFLAGS) yylex$(CIEXT)
- XX
- XXflex.man : flex.1
- XX nroff -man flex.1 >flex.man
- XX
- XXlint : $(FLEX_C_SOURCES)
- XX lint $(FLEX_C_SOURCES) > flex.lint
- XX
- XXdistrib :
- XX mv scan.c initscan.c
- XX chmod 444 initscan.c
- XX $(MAKE) $(MFLAGS) clean
- XX
- XXclean :
- XX rm -f core errs flex *$(OEXT) parse.c *.lint parse.h flex.man tags
- XX
- XXtags :
- XX ctags $(FLEX_C_SOURCES)
- XX
- XXvms : flex.man
- XX $(MAKE) $(MFLAGS) distrib
- XX
- XXtest :
- XX $(FLEX) $(FLEX_FLAGS) scan.l
- XX $(RENAME) lex_yy.c scan.ctest
- XX cmp scan.c scan.ctest
- XX
- X/
- Xecho x - Readme2
- Xsed '/^X/s///' > Readme2 << '/'
- XXThe changes: (1.7.89 RAL)
- XX
- XX - Bug fix: The original flex didn't like trailing spaces in exclusive start
- XX condition lists ! If you add an trailing space to line 68 in scan.l
- XX
- XX "%x SECT2 SECT2PROLOG SECT3 CODEBLOCK PICKUPDEF SC CARETISBOL NUM QUOTE "
- XX
- XX you get a misleading error message:
- XX
- XX "Syntax error at line 69: bad start condition list"
- XX
- XX This bug can either be fixed in parse.y or in scan.l . I have chosen the
- XX last because there the fix is minimal: Just change the rule (line 128)
- XX
- XX "\n" to "{OPTWS}\n"
- XX
- XX - Enhancements:
- XX - new option "-?" that provides some help information about the other
- XX flags (main.c)
- XX - new option "-aTMPPATH" that allows a redefinition of the standard
- XX path for the temporary file (this might be rather large >200k if
- XX F,f options are selected). (main.c, flexdef.h (l.376))
- XX - hexdump of illegal characters -- this proved to be a useful debugging
- XX tool especialy if invisible control characters occur which weren't
- XX covered by the rules. (scan.l fprintf statement line 129,...)
- XX
- XX - Patches due to TOS
- XX - General: TOS uses CR,LF as end of line char., Flex wants only a single
- XX LF as EOL char. Therefore all I/O must be translated using f* calls.
- XX This is done everywhere besides the YY_INPUT macro (flex.skel (scan.c),
- XX line 31) that uses a low level 'read'. This should be definitly changed
- XX to fread, so that all I/O calls occur on the same level.
- XX - the short action_file_name has been "flexXXXXXX.tmp", but that's too
- XX much for TOS,MSDOS ! I changed it to "flexXXXX.tmp" in main.c (patch
- XX covered by the -a option additions)
- XX - some compilers don't like external names that are ambiguous within
- XX the first 8 characters. I defined macros that change all these long
- XX external names to names that are unique within the first 8 characters.
- XX Just define SHORT_EXTERNAL_NAMES to use this feature (flexdef.h)
- XX - some statement changes that some compiler don't like:
- XX typedef *xxx[] -> typedef **xxx (flexdef.h.,l.308)
- XX "/*" -> '/','*' within a comment in (scan.l, l.209)
- XX - changed short "lexyy.c" to "lex_yy.c" what's more similar to the unix
- XX "lex.yy.c" (main.c).
- XX - a few further really compiler dependent changes provided with
- XX #ifdef ATARI && LATTICE res. TURBO braces.
- XX
- XX - Additional TOS only files
- XX - Makefile.tos: Common makefile for all TOS compilers. If you add further
- XX ones please email me the new makefile.
- XX - flex.lnk: Lattice - GST linker argument extension file
- XX - flex.tlk: Turbo linker argument extension file
- XX
- XX
- XXAdditional remarks:
- XX
- XXI didn't add a new initscan.c (= flex -ist scan.l). The current one is good
- XXenough for a first compilation. With this first version of flex you can
- XXrebuild your own scan.c and the final flex version !
- XX
- XXMinix ST :
- XX - I had to "chmem =70000 cv" (>50000) to get flex linked
- XX - 'memset' (PC 1.3, EFTH40,...) is necessary
- XX - chmem =90000 flex may be sufficient
- XX
- XXMinix PC :
- XX It should be possible to port Flex to Minix PC. The current sizes of flex
- XX are:
- XX Minix ST (ACK) Lattice (TOS) Turbo (TOS)
- XX
- XX size 75300 83305 57957
- XX compilation time 22' 15' 3'40"
- XX flex -is scan.l 1'49" 43" 30"
- XX
- XX The Minix ST size includes the bad generated code using only a subset of
- XX the 68000 commands, long addresses only and a huge relocation table.
- XX Therefore the PC size will be <64 k ! More serious is the fact that I had
- XX to chmem =90000 flex to get scan.l converted to scan.c . But I never saw
- XX a more complex lex source than scan.l -- so it should be possible to
- XX reduce some array sizes without limitation for all day usage.
- XX
- XX No one volunteered yet for a Minix PC port -- but if someone will try it
- XX I would provide him with a new scan.c and some hints.
- XX
- XXTOS:
- XX Don't forget to adapt the flexskel path within flexdef.h !
- XX
- XX
- X/
- Xecho x - flex.lnk
- Xsed '/^X/s///' > flex.lnk << '/'
- XX*
- XX*
- XX* linker control file for flex.ttp
- XX*
- XX*
- XX*
- XXINPUT d:\latt\lib\startup.bin
- XX*
- XXINPUT ccl.bin
- XXINPUT dfa.bin
- XXINPUT ecs.bin
- XXINPUT gen.bin
- XXINPUT misc.bin
- XXINPUT nfa.bin
- XXINPUT parse.bin
- XXINPUT sym.bin
- XXINPUT tblcmp.bin
- XXINPUT main.bin
- XXINPUT yylex.bin
- XXINPUT scan.bin
- XX*
- XXLIBRARY d:\latt\lib\clib.bin
- XX
- X/
- Xecho x - flex.tlk
- Xsed '/^X/s///' > flex.tlk << '/'
- XXccl.o
- XXdfa.o
- XXecs.o
- XXgen.o
- XXmisc.o
- XXnfa.o
- XXparse.o
- XXsym.o
- XXtblcmp.o
- XXyylex.o
- XXmain.o
- XXscan.o
- XXd:\turbo\lib\tcstdlib.lib ; standard lib
- XXd:\turbo\lib\tcextlib.lib ; extended lib
- XXd:\turbo\lib\tctoslib.lib ; TOS lib
- XXd:\turbo\lib\tcgemlib.lib ; AES and VDI lib
- XX-S=200000
- X/
- END_OF_FILE
- if test 29474 -ne `wc -c <'MISC/Atari.patches'`; then
- echo shar: \"'MISC/Atari.patches'\" unpacked with wrong size!
- fi
- # end of 'MISC/Atari.patches'
- fi
- if test -f 'flex.1' -a "${1}" != "-c" ; then
- echo shar: Will not clobber existing file \"'flex.1'\"
- else
- echo shar: Extracting \"'flex.1'\" \(20799 characters\)
- sed "s/^X//" >'flex.1' <<'END_OF_FILE'
- X.TH FLEX 1 "26 May 1990" "Version 2.3"
- X.SH NAME
- Xflex - fast lexical analyzer generator
- X.SH SYNOPSIS
- X.B flex
- X.B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton]
- X.I [filename ...]
- X.SH DESCRIPTION
- X.I flex
- Xis a tool for generating
- X.I scanners:
- Xprograms which recognized lexical patterns in text.
- X.I flex
- Xreads
- Xthe given input files, or its standard input if no file names are given,
- Xfor a description of a scanner to generate. The description is in
- Xthe form of pairs
- Xof regular expressions and C code, called
- X.I rules. flex
- Xgenerates as output a C source file,
- X.B lex.yy.c,
- Xwhich defines a routine
- X.B yylex().
- XThis file is compiled and linked with the
- X.B -lfl
- Xlibrary to produce an executable. When the executable is run,
- Xit analyzes its input for occurrences
- Xof the regular expressions. Whenever it finds one, it executes
- Xthe corresponding C code.
- X.LP
- XFor full documentation, see
- X.B flexdoc(1).
- XThis manual entry is intended for use as a quick reference.
- X.SH OPTIONS
- X.I flex
- Xhas the following options:
- X.TP
- X.B -b
- XGenerate backtracking information to
- X.I lex.backtrack.
- XThis is a list of scanner states which require backtracking
- Xand the input characters on which they do so. By adding rules one
- Xcan remove backtracking states. If all backtracking states
- Xare eliminated and
- X.B -f
- Xor
- X.B -F
- Xis used, the generated scanner will run faster.
- X.TP
- X.B -c
- Xis a do-nothing, deprecated option included for POSIX compliance.
- X.IP
- X.B NOTE:
- Xin previous releases of
- X.I flex
- X.B -c
- Xspecified table-compression options. This functionality is
- Xnow given by the
- X.B -C
- Xflag. To ease the the impact of this change, when
- X.I flex
- Xencounters
- X.B -c,
- Xit currently issues a warning message and assumes that
- X.B -C
- Xwas desired instead. In the future this "promotion" of
- X.B -c
- Xto
- X.B -C
- Xwill go away in the name of full POSIX compliance (unless
- Xthe POSIX meaning is removed first).
- X.TP
- X.B -d
- Xmakes the generated scanner run in
- X.I debug
- Xmode. Whenever a pattern is recognized and the global
- X.B yy_flex_debug
- Xis non-zero (which is the default), the scanner will
- Xwrite to
- X.I stderr
- Xa line of the form:
- X.nf
- X
- X --accepting rule at line 53 ("the matched text")
- X
- X.fi
- XThe line number refers to the location of the rule in the file
- Xdefining the scanner (i.e., the file that was fed to flex). Messages
- Xare also generated when the scanner backtracks, accepts the
- Xdefault rule, reaches the end of its input buffer (or encounters
- Xa NUL; the two look the same as far as the scanner's concerned),
- Xor reaches an end-of-file.
- X.TP
- X.B -f
- Xspecifies (take your pick)
- X.I full table
- Xor
- X.I fast scanner.
- XNo table compression is done. The result is large but fast.
- XThis option is equivalent to
- X.B -Cf
- X(see below).
- X.TP
- X.B -i
- Xinstructs
- X.I flex
- Xto generate a
- X.I case-insensitive
- Xscanner. The case of letters given in the
- X.I flex
- Xinput patterns will
- Xbe ignored, and tokens in the input will be matched regardless of case. The
- Xmatched text given in
- X.I yytext
- Xwill have the preserved case (i.e., it will not be folded).
- X.TP
- X.B -n
- Xis another do-nothing, deprecated option included only for
- XPOSIX compliance.
- X.TP
- X.B -p
- Xgenerates a performance report to stderr. The report
- Xconsists of comments regarding features of the
- X.I flex
- Xinput file which will cause a loss of performance in the resulting scanner.
- X.TP
- X.B -s
- Xcauses the
- X.I default rule
- X(that unmatched scanner input is echoed to
- X.I stdout)
- Xto be suppressed. If the scanner encounters input that does not
- Xmatch any of its rules, it aborts with an error.
- X.TP
- X.B -t
- Xinstructs
- X.I flex
- Xto write the scanner it generates to standard output instead
- Xof
- X.B lex.yy.c.
- X.TP
- X.B -v
- Xspecifies that
- X.I flex
- Xshould write to
- X.I stderr
- Xa summary of statistics regarding the scanner it generates.
- X.TP
- X.B -F
- Xspecifies that the
- X.ul
- Xfast
- Xscanner table representation should be used. This representation is
- Xabout as fast as the full table representation
- X.ul
- X(-f),
- Xand for some sets of patterns will be considerably smaller (and for
- Xothers, larger). See
- X.B flexdoc(1)
- Xfor details.
- X.IP
- XThis option is equivalent to
- X.B -CF
- X(see below).
- X.TP
- X.B -I
- Xinstructs
- X.I flex
- Xto generate an
- X.I interactive
- Xscanner, that is, a scanner which stops immediately rather than
- Xlooking ahead if it knows
- Xthat the currently scanned text cannot be part of a longer rule's match.
- XAgain, see
- X.B flexdoc(1)
- Xfor details.
- X.IP
- XNote,
- X.B -I
- Xcannot be used in conjunction with
- X.I full
- Xor
- X.I fast tables,
- Xi.e., the
- X.B -f, -F, -Cf,
- Xor
- X.B -CF
- Xflags.
- X.TP
- X.B -L
- Xinstructs
- X.I flex
- Xnot to generate
- X.B #line
- Xdirectives in
- X.B lex.yy.c.
- XThe default is to generate such directives so error
- Xmessages in the actions will be correctly
- Xlocated with respect to the original
- X.I flex
- Xinput file, and not to
- Xthe fairly meaningless line numbers of
- X.B lex.yy.c.
- X.TP
- X.B -T
- Xmakes
- X.I flex
- Xrun in
- X.I trace
- Xmode. It will generate a lot of messages to
- X.I stdout
- Xconcerning
- Xthe form of the input and the resultant non-deterministic and deterministic
- Xfinite automata. This option is mostly for use in maintaining
- X.I flex.
- X.TP
- X.B -8
- Xinstructs
- X.I flex
- Xto generate an 8-bit scanner.
- XOn some sites, this is the default. On others, the default
- Xis 7-bit characters. To see which is the case, check the verbose
- X.B (-v)
- Xoutput for "equivalence classes created". If the denominator of
- Xthe number shown is 128, then by default
- X.I flex
- Xis generating 7-bit characters. If it is 256, then the default is
- X8-bit characters.
- X.TP
- X.B -C[efmF]
- Xcontrols the degree of table compression.
- X.IP
- X.B -Ce
- Xdirects
- X.I flex
- Xto construct
- X.I equivalence classes,
- Xi.e., sets of characters
- Xwhich have identical lexical properties.
- XEquivalence classes usually give
- Xdramatic reductions in the final table/object file sizes (typically
- Xa factor of 2-5) and are pretty cheap performance-wise (one array
- Xlook-up per character scanned).
- X.IP
- X.B -Cf
- Xspecifies that the
- X.I full
- Xscanner tables should be generated -
- X.I flex
- Xshould not compress the
- Xtables by taking advantages of similar transition functions for
- Xdifferent states.
- X.IP
- X.B -CF
- Xspecifies that the alternate fast scanner representation (described in
- X.B flexdoc(1))
- Xshould be used.
- X.IP
- X.B -Cm
- Xdirects
- X.I flex
- Xto construct
- X.I meta-equivalence classes,
- Xwhich are sets of equivalence classes (or characters, if equivalence
- Xclasses are not being used) that are commonly used together. Meta-equivalence
- Xclasses are often a big win when using compressed tables, but they
- Xhave a moderate performance impact (one or two "if" tests and one
- Xarray look-up per character scanned).
- X.IP
- XA lone
- X.B -C
- Xspecifies that the scanner tables should be compressed but neither
- Xequivalence classes nor meta-equivalence classes should be used.
- X.IP
- XThe options
- X.B -Cf
- Xor
- X.B -CF
- Xand
- X.B -Cm
- Xdo not make sense together - there is no opportunity for meta-equivalence
- Xclasses if the table is not being compressed. Otherwise the options
- Xmay be freely mixed.
- X.IP
- XThe default setting is
- X.B -Cem,
- Xwhich specifies that
- X.I flex
- Xshould generate equivalence classes
- Xand meta-equivalence classes. This setting provides the highest
- Xdegree of table compression. You can trade off
- Xfaster-executing scanners at the cost of larger tables with
- Xthe following generally being true:
- X.nf
- X
- X slowest & smallest
- X -Cem
- X -Cm
- X -Ce
- X -C
- X -C{f,F}e
- X -C{f,F}
- X fastest & largest
- X
- X.fi
- X.IP
- X.B -C
- Xoptions are not cumulative; whenever the flag is encountered, the
- Xprevious -C settings are forgotten.
- X.TP
- X.B -Sskeleton_file
- Xoverrides the default skeleton file from which
- X.I flex
- Xconstructs its scanners. You'll never need this option unless you are doing
- X.I flex
- Xmaintenance or development.
- X.SH SUMMARY OF FLEX REGULAR EXPRESSIONS
- XThe patterns in the input are written using an extended set of regular
- Xexpressions. These are:
- X.nf
- X
- X x match the character 'x'
- X . any character except newline
- X [xyz] a "character class"; in this case, the pattern
- X matches either an 'x', a 'y', or a 'z'
- X [abj-oZ] a "character class" with a range in it; matches
- X an 'a', a 'b', any letter from 'j' through 'o',
- X or a 'Z'
- X [^A-Z] a "negated character class", i.e., any character
- X but those in the class. In this case, any
- X character EXCEPT an uppercase letter.
- X [^A-Z\\n] any character EXCEPT an uppercase letter or
- X a newline
- X r* zero or more r's, where r is any regular expression
- X r+ one or more r's
- X r? zero or one r's (that is, "an optional r")
- X r{2,5} anywhere from two to five r's
- X r{2,} two or more r's
- X r{4} exactly 4 r's
- X {name} the expansion of the "name" definition
- X (see above)
- X "[xyz]\\"foo"
- X the literal string: [xyz]"foo
- X \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
- X then the ANSI-C interpretation of \\x.
- X Otherwise, a literal 'X' (used to escape
- X operators such as '*')
- X \\123 the character with octal value 123
- X \\x2a the character with hexadecimal value 2a
- X (r) match an r; parentheses are used to override
- X precedence (see below)
- X
- X
- X rs the regular expression r followed by the
- X regular expression s; called "concatenation"
- X
- X
- X r|s either an r or an s
- X
- X
- X r/s an r but only if it is followed by an s. The
- X s is not part of the matched text. This type
- X of pattern is called as "trailing context".
- X ^r an r, but only at the beginning of a line
- X r$ an r, but only at the end of a line. Equivalent
- X to "r/\\n".
- X
- X
- X <s>r an r, but only in start condition s (see
- X below for discussion of start conditions)
- X <s1,s2,s3>r
- X same, but in any of start conditions s1,
- X s2, or s3
- X
- X
- X <<EOF>> an end-of-file
- X <s1,s2><<EOF>>
- X an end-of-file when in start condition s1 or s2
- X
- X.fi
- XThe regular expressions listed above are grouped according to
- Xprecedence, from highest precedence at the top to lowest at the bottom.
- XThose grouped together have equal precedence.
- X.LP
- XSome notes on patterns:
- X.IP -
- XNegated character classes
- X.I match newlines
- Xunless "\\n" (or an equivalent escape sequence) is one of the
- Xcharacters explicitly present in the negated character class
- X(e.g., "[^A-Z\\n]").
- X.IP -
- XA rule can have at most one instance of trailing context (the '/' operator
- Xor the '$' operator). The start condition, '^', and "<<EOF>>" patterns
- Xcan only occur at the beginning of a pattern, and, as well as with '/' and '$',
- Xcannot be grouped inside parentheses. The following are all illegal:
- X.nf
- X
- X foo/bar$
- X foo|(bar$)
- X foo|^bar
- X <sc1>foo<sc2>bar
- X
- X.fi
- X.SH SUMMARY OF SPECIAL ACTIONS
- XIn addition to arbitrary C code, the following can appear in actions:
- X.IP -
- X.B ECHO
- Xcopies yytext to the scanner's output.
- X.IP -
- X.B BEGIN
- Xfollowed by the name of a start condition places the scanner in the
- Xcorresponding start condition.
- X.IP -
- X.B REJECT
- Xdirects the scanner to proceed on to the "second best" rule which matched the
- Xinput (or a prefix of the input).
- X.B yytext
- Xand
- X.B yyleng
- Xare set up appropriately. Note that
- X.B REJECT
- Xis a particularly expensive feature in terms scanner performance;
- Xif it is used in
- X.I any
- Xof the scanner's actions it will slow down
- X.I all
- Xof the scanner's matching. Furthermore,
- X.B REJECT
- Xcannot be used with the
- X.I -f
- Xor
- X.I -F
- Xoptions.
- X.IP
- XNote also that unlike the other special actions,
- X.B REJECT
- Xis a
- X.I branch;
- Xcode immediately following it in the action will
- X.I not
- Xbe executed.
- X.IP -
- X.B yymore()
- Xtells the scanner that the next time it matches a rule, the corresponding
- Xtoken should be
- X.I appended
- Xonto the current value of
- X.B yytext
- Xrather than replacing it.
- X.IP -
- X.B yyless(n)
- Xreturns all but the first
- X.I n
- Xcharacters of the current token back to the input stream, where they
- Xwill be rescanned when the scanner looks for the next match.
- X.B yytext
- Xand
- X.B yyleng
- Xare adjusted appropriately (e.g.,
- X.B yyleng
- Xwill now be equal to
- X.I n
- X).
- X.IP -
- X.B unput(c)
- Xputs the character
- X.I c
- Xback onto the input stream. It will be the next character scanned.
- X.IP -
- X.B input()
- Xreads the next character from the input stream (this routine is called
- X.B yyinput()
- Xif the scanner is compiled using
- X.B C++).
- X.IP -
- X.B yyterminate()
- Xcan be used in lieu of a return statement in an action. It terminates
- Xthe scanner and returns a 0 to the scanner's caller, indicating "all done".
- X.IP
- XBy default,
- X.B yyterminate()
- Xis also called when an end-of-file is encountered. It is a macro and
- Xmay be redefined.
- X.IP -
- X.B YY_NEW_FILE
- Xis an action available only in <<EOF>> rules. It means "Okay, I've
- Xset up a new input file, continue scanning".
- X.IP -
- X.B yy_create_buffer( file, size )
- Xtakes a
- X.I FILE
- Xpointer and an integer
- X.I size.
- XIt returns a YY_BUFFER_STATE
- Xhandle to a new input buffer large enough to accomodate
- X.I size
- Xcharacters and associated with the given file. When in doubt, use
- X.B YY_BUF_SIZE
- Xfor the size.
- X.IP -
- X.B yy_switch_to_buffer( new_buffer )
- Xswitches the scanner's processing to scan for tokens from
- Xthe given buffer, which must be a YY_BUFFER_STATE.
- X.IP -
- X.B yy_delete_buffer( buffer )
- Xdeletes the given buffer.
- X.SH VALUES AVAILABLE TO THE USER
- X.IP -
- X.B char *yytext
- Xholds the text of the current token. It may not be modified.
- X.IP -
- X.B int yyleng
- Xholds the length of the current token. It may not be modified.
- X.IP -
- X.B FILE *yyin
- Xis the file which by default
- X.I flex
- Xreads from. It may be redefined but doing so only makes sense before
- Xscanning begins. Changing it in the middle of scanning will have
- Xunexpected results since
- X.I flex
- Xbuffers its input. Once scanning terminates because an end-of-file
- Xhas been seen,
- X.B
- Xvoid yyrestart( FILE *new_file )
- Xmay be called to point
- X.I yyin
- Xat the new input file.
- X.IP -
- X.B FILE *yyout
- Xis the file to which
- X.B ECHO
- Xactions are done. It can be reassigned by the user.
- X.IP -
- X.B YY_CURRENT_BUFFER
- Xreturns a
- X.B YY_BUFFER_STATE
- Xhandle to the current buffer.
- X.SH MACROS THE USER CAN REDEFINE
- X.IP -
- X.B YY_DECL
- Xcontrols how the scanning routine is declared.
- XBy default, it is "int yylex()", or, if prototypes are being
- Xused, "int yylex(void)". This definition may be changed by redefining
- Xthe "YY_DECL" macro. Note that
- Xif you give arguments to the scanning routine using a
- XK&R-style/non-prototyped function declaration, you must terminate
- Xthe definition with a semi-colon (;).
- X.IP -
- XThe nature of how the scanner
- Xgets its input can be controlled by redefining the
- X.B YY_INPUT
- Xmacro.
- XYY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
- Xaction is to place up to
- X.I max_size
- Xcharacters in the character array
- X.I buf
- Xand return in the integer variable
- X.I result
- Xeither the
- Xnumber of characters read or the constant YY_NULL (0 on Unix systems)
- Xto indicate EOF. The default YY_INPUT reads from the
- Xglobal file-pointer "yyin".
- XA sample redefinition of YY_INPUT (in the definitions
- Xsection of the input file):
- X.nf
- X
- X %{
- X #undef YY_INPUT
- X #define YY_INPUT(buf,result,max_size) \\
- X { \\
- X int c = getchar(); \\
- X result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
- X }
- X %}
- X
- X.fi
- X.IP -
- XWhen the scanner receives an end-of-file indication from YY_INPUT,
- Xit then checks the
- X.B yywrap()
- Xfunction. If
- X.B yywrap()
- Xreturns false (zero), then it is assumed that the
- Xfunction has gone ahead and set up
- X.I yyin
- Xto point to another input file, and scanning continues. If it returns
- Xtrue (non-zero), then the scanner terminates, returning 0 to its
- Xcaller.
- X.IP
- XThe default
- X.B yywrap()
- Xalways returns 1. Presently, to redefine it you must first
- X"#undef yywrap", as it is currently implemented as a macro. It is
- Xlikely that
- X.B yywrap()
- Xwill soon be defined to be a function rather than a macro.
- X.IP -
- XYY_USER_ACTION
- Xcan be redefined to provide an action
- Xwhich is always executed prior to the matched rule's action.
- X.IP -
- XThe macro
- X.B YY_USER_INIT
- Xmay be redefined to provide an action which is always executed before
- Xthe first scan.
- X.IP -
- XIn the generated scanner, the actions are all gathered in one large
- Xswitch statement and separated using
- X.B YY_BREAK,
- Xwhich may be redefined. By default, it is simply a "break", to separate
- Xeach rule's action from the following rule's.
- X.SH FILES
- X.TP
- X.I flex.skel
- Xskeleton scanner.
- X.TP
- X.I lex.yy.c
- Xgenerated scanner (called
- X.I lexyy.c
- Xon some systems).
- X.TP
- X.I lex.backtrack
- Xbacktracking information for
- X.B -b
- Xflag (called
- X.I lex.bck
- Xon some systems).
- X.TP
- X.B -lfl
- Xlibrary with which to link the scanners.
- X.SH "SEE ALSO"
- X.LP
- Xflexdoc(1), lex(1), yacc(1), sed(1), awk(1).
- X.LP
- XM. E. Lesk and E. Schmidt,
- X.I LEX - Lexical Analyzer Generator
- X.SH DIAGNOSTICS
- X.I reject_used_but_not_detected undefined
- Xor
- X.LP
- X.I yymore_used_but_not_detected undefined -
- XThese errors can occur at compile time. They indicate that the
- Xscanner uses
- X.B REJECT
- Xor
- X.B yymore()
- Xbut that
- X.I flex
- Xfailed to notice the fact, meaning that
- X.I flex
- Xscanned the first two sections looking for occurrences of these actions
- Xand failed to find any, but somehow you snuck some in (via a #include
- Xfile, for example). Make an explicit reference to the action in your
- X.I flex
- Xinput file. (Note that previously
- X.I flex
- Xsupported a
- X.B %used/%unused
- Xmechanism for dealing with this problem; this feature is still supported
- Xbut now deprecated, and will go away soon unless the author hears from
- Xpeople who can argue compellingly that they need it.)
- X.LP
- X.I flex scanner jammed -
- Xa scanner compiled with
- X.B -s
- Xhas encountered an input string which wasn't matched by
- Xany of its rules.
- X.LP
- X.I flex input buffer overflowed -
- Xa scanner rule matched a string long enough to overflow the
- Xscanner's internal input buffer (16K bytes - controlled by
- X.B YY_BUF_MAX
- Xin "flex.skel").
- X.LP
- X.I scanner requires -8 flag -
- XYour scanner specification includes recognizing 8-bit characters and
- Xyou did not specify the -8 flag (and your site has not installed flex
- Xwith -8 as the default).
- X.LP
- X.I
- Xfatal flex scanner internal error--end of buffer missed -
- XThis can occur in an scanner which is reentered after a long-jump
- Xhas jumped out (or over) the scanner's activation frame. Before
- Xreentering the scanner, use:
- X.nf
- X
- X yyrestart( yyin );
- X
- X.fi
- X.LP
- X.I too many %t classes! -
- XYou managed to put every single character into its own %t class.
- X.I flex
- Xrequires that at least one of the classes share characters.
- X.SH AUTHOR
- XVern Paxson, with the help of many ideas and much inspiration from
- XVan Jacobson. Original version by Jef Poskanzer.
- X.LP
- XSee flexdoc(1) for additional credits and the address to send comments to.
- X.SH DEFICIENCIES / BUGS
- X.LP
- XSome trailing context
- Xpatterns cannot be properly matched and generate
- Xwarning messages ("Dangerous trailing context"). These are
- Xpatterns where the ending of the
- Xfirst part of the rule matches the beginning of the second
- Xpart, such as "zx*/xy*", where the 'x*' matches the 'x' at
- Xthe beginning of the trailing context. (Note that the POSIX draft
- Xstates that the text matched by such patterns is undefined.)
- X.LP
- XFor some trailing context rules, parts which are actually fixed-length are
- Xnot recognized as such, leading to the abovementioned performance loss.
- XIn particular, parts using '|' or {n} (such as "foo{3}") are always
- Xconsidered variable-length.
- X.LP
- XCombining trailing context with the special '|' action can result in
- X.I fixed
- Xtrailing context being turned into the more expensive
- X.I variable
- Xtrailing context. For example, this happens in the following example:
- X.nf
- X
- X %%
- X abc |
- X xyz/def
- X
- X.fi
- X.LP
- XUse of unput() invalidates yytext and yyleng.
- X.LP
- XUse of unput() to push back more text than was matched can
- Xresult in the pushed-back text matching a beginning-of-line ('^')
- Xrule even though it didn't come at the beginning of the line
- X(though this is rare!).
- X.LP
- XPattern-matching of NUL's is substantially slower than matching other
- Xcharacters.
- X.LP
- X.I flex
- Xdoes not generate correct #line directives for code internal
- Xto the scanner; thus, bugs in
- X.I flex.skel
- Xyield bogus line numbers.
- X.LP
- XDue to both buffering of input and read-ahead, you cannot intermix
- Xcalls to <stdio.h> routines, such as, for example,
- X.B getchar(),
- Xwith
- X.I flex
- Xrules and expect it to work. Call
- X.B input()
- Xinstead.
- X.LP
- XThe total table entries listed by the
- X.B -v
- Xflag excludes the number of table entries needed to determine
- Xwhat rule has been matched. The number of entries is equal
- Xto the number of DFA states if the scanner does not use
- X.B REJECT,
- Xand somewhat greater than the number of states if it does.
- X.LP
- X.B REJECT
- Xcannot be used with the
- X.I -f
- Xor
- X.I -F
- Xoptions.
- X.LP
- XSome of the macros, such as
- X.B yywrap(),
- Xmay in the future become functions which live in the
- X.B -lfl
- Xlibrary. This will doubtless break a lot of code, but may be
- Xrequired for POSIX-compliance.
- X.LP
- XThe
- X.I flex
- Xinternal algorithms need documentation.
- END_OF_FILE
- if test 20799 -ne `wc -c <'flex.1'`; then
- echo shar: \"'flex.1'\" unpacked with wrong size!
- fi
- # end of 'flex.1'
- fi
- echo shar: End of archive 4 \(of 10\).
- cp /dev/null ark4isdone
- MISSING=""
- for I in 1 2 3 4 5 6 7 8 9 10 ; do
- if test ! -f ark${I}isdone ; then
- MISSING="${MISSING} ${I}"
- fi
- done
- if test "${MISSING}" = "" ; then
- echo You have unpacked all 10 archives.
- rm -f ark[1-9]isdone ark[1-9][0-9]isdone
- else
- echo You still must unpack the following archives:
- echo " " ${MISSING}
- fi
- exit 0
- exit 0 # Just in case...
-